[GDPVal] Add model-agnostic Slurm orchestration for multi-allocation rollouts - #2162

Reference scripts for collecting GDPVal rollouts on a Slurm cluster with a
self-hosted vLLM endpoint, across more wall-clock time than a single allocation
provides.

They exist because a GDPVal rollout batch does not fit in one job. Tasks run for
tens of minutes each, allocations are time-boxed, and a run that is merely
restarted after an allocation expires silently loses or duplicates work. These
keep the model server and the rollout client in separate jobs, hand off between
allocations, and refuse to resume when the state is not provably consistent.

Nothing here is required to run GDPVal — gym eval run is. This is for batches
too large for one allocation.

Configuration is three independent layers

Serving a different model is a new file in models/, not an edit to any script.
run_parallel_vllm.sh builds its vllm serve invocation from the profile —
TP/PP/DP, GPU memory, context, --tool-call-parser, --reasoning-parser,
--kv-cache-dtype, expert-parallel, enforce-eager — plus a
GDPVAL_VLLM_EXTRA_ARGS passthrough so a new model never requires touching the
launcher. models/glm-5.2-fp8.env and glm-5.2-bf16.env ship as worked
examples: the same weights shaped for latency (2 nodes, TP=8, fp8 KV cache)
versus throughput (16 nodes, TP=4, DP=16).

Required settings have no defaults

GDPVAL_SLURM_ACCOUNT, GDPVAL_MODEL_PATH, GDPVAL_VLLM_IMAGE and
GDPVAL_MODEL_BASE_NAME exit 2 with the variable name when unset. A default
that looks plausible but points at the wrong model or account does not fail at
submit time — it fails hours later as bad numbers, which is far more expensive
than an error message.

Resume safety

validate_gdpval_resume_state.py gates every continuation: no duplicate or
missing task indices, and launch/source files unchanged since the run started.
Editing the launcher or validator mid-run changes their hashes and the gate
refuses to resume. That is deliberate — silently resuming against changed code
produces a batch that is two experiments stitched together.

Verification

CI does not exercise these: there is no shellcheck hook and no test covers
benchmarks/gdpval/slurm/, so a green tick here means the rest of the repo is
unaffected, not that the scripts work. What they have behind them is bash -n
on all 13 scripts, py_compile on the validator, and a completed 200-task
GDPVal rollout collection driven by them end to end across 7 allocations.

Job shapes reflect the one cluster they were built and run on. Partition names,
QOS and GPUs per node are configurable, but expect to adjust cluster.env for
yours.